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ABSTRACT 

This paper reports on an ongoing effort of the Vermont Mental 
Health Performance Indicator Project (PIP) to examine the relevance and 
utility of standardized test scores for evaluating community mental health 
programs. This analysis is of test scores from Vermont’s first four years of 
statewide testing. The study is examining anonymous person-level extracts 
from four existing databases: the Mathematics Skills Assessment and the 
English Language Reading — Basic Understanding for 4th-, 8th-, and lOth-grade 
students during 1998 through 2001; the Vermont Mental Health Division’s 
Monthly Service Report; and the Vermont District Court records. For mental 
health service recipients, performance by boys on the mathematics test 
increased, whereas the performance of girls decreased. This resulted, 
however, in similar lOth-grade performance for both boys and girls. 
Performance on the English test decreased between 8th and 10th grades for 
both boys and girls in the treatment group. Involvement with the criminal 
justice system was substantially greater for students who did not participate 
in the tests than for participants. Results support the value of educational 
test scores for evaluating systems of care for children and adolescents and 
for providing valid and reliable information on levels of school 
participation and school performance for recipients of mental health 
services. (DB) 
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Introduction 



John A. Pandiani 
Monica M. Simon 
Steven M. Banks 



Standardized educational test scores are becoming an increasingly 
important part of the childrens services data infrastructure. The recently reauthorized Elementary and 
vSecondary Education Act (also known as the No Child Left Behind Act, 2001) requires statewide 
reading and mathematics tests each year for children in grades three through eight by 2005-06. This 
legislation follows efforts in many states to use standardized testing to introduce higher levels of 
accountability in education. The Vermont Department of Education implemented a statewide, 
standardized testing program, the New Standards Reference Exams, in 1998. Since that time, public 
school students throughout the state have been tested on an annual basis in mathematics and English/ 
language at the fourth, eighth, and tenth grade levels. Vermont’s standardized tests are not used to 
determine student advancement, but are used as a core measure of school performance. 



This proliferation of standardized testing is occurring at a time when children’s mental health 
programs and service systems are being increasingly called upon to demonstrate their favorable impact 
on school participation and performance (National Association of State Mental Health Programs 
Directors, 1998; Rosenblatt, 1998). Within this context, the Vermont Mental Health Performance 
Indicator Project (PIP) is exploring the relevance and utility of standardized test scores for evaluating 
community mental health programs. This paper reports on the results of a preliminary analysis of 
standardized test scores from Vermont’s first four years of statewide testing. 



Method 

Data 

The findings reported here are based exclusively on analysis of anonymous extracts from four 
existing databases. The anonymous, person-level extracts from the Department of Education’s 
Mathematics Skills Assessment, and the English Language Reading: Basic Understanding, databases for 
4th, 8th, and 10th grade students during 1998 through 2001 provided our measures of school 
participation and performance. The number of young people represented in these education data sets 
averaged 20,743 per year. 

Third, anonymous, person-level extracts from the Vermont Mental Health Division’s Monthly 
Service Report database provided basic information on all young people who received community 
mental health services during 1998 through 2000. The number of young people in the relevant age 
groups who were served by community mental health programs averaged 2,279 per year. 

Fourth, anonymous event level extracts from the Vermont District Court database provided basic 
information on all individuals who were charged with a crime in Vermont during 1998 through 2001. 
The number of young people in the relevant age groups who were charged with a crime set averaged 
1 ,599 per year. 

Analysis 

School participation is demonstrated by completing the standardized test(s) on grade level or one 
year later. Operationally, school participation rates for mental health clients are the overlap between 
the mental health and the school test data set. School participation rates for all young people were 
obtained by comparing the total number of young people tested to the total population of the state. 
Our measure oi school performance is the proportion of students who score at or above the “standard” 
for the test. Operationally, school performance rates for mental health clients are the overlap between 
the mental health data set and a data set of “high” test scorers. 
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The relationship between these school-based measures and other outcomes was measured by 
comparing participation/performance rates to rates of being charged with a crime for 16 year old 
students who scored at or above standard, who scored below standard, and who were not tested (on 
grade level). 

Because the data sets used in this analysis do not share unique person identifiers. Probabilistic 
Population Estimation was used to determine the number of people who appeared in combinations of 
data sets. Probabilistic Population Estimation is a statistical procedure that provides valid and reliable 
measures of the size and overlap of data sets that do not include unique person identifiers (Banks & 
Pandiani, 2001). These estimates arc based on comparisons of the distribution of dates of birth in the 
data sets to the known distribution of dates of birth in the general population. 

Findings 

Recipients of community mental health services were less likely to take part in Vermont’s statewide 
school testing than other students. Statewide, only 41% of the mental health service recipients 
participated in the Mathematics Skills Assessment, compared to 67% of all students. Female service 
recipients were more likely to participate in school testing than were male service recipients in the 4th 
and 8th grade, but this difference disappeared in the 10th grade. These patterns were evident for both 
mathematics and English tests. There were also significant differences among Vermont’s regional 
community mental health programs in relative test participation (controlling for general population 
participation rates). 

Recipients of community mental health services were less likely than other students to score at or 
above standard on Vermont’s standardized school tests. Only 42% of the mental health service 
recipients scored at or above standard on the mathematics test, compared to 64% of other students, 
and 49% of service recipients scored at or above standard on the English test compared to 65% of 
other students. There were no statistically significant differences between age or gender groups in 
mathematics test performance among service recipients or in the general population. Service 
recipients’ performance on the standardized tests in some regions of the state, however, was similar to 
the performance of other students. 

When change in performance is examined, results for service recipients and other students are 
similar. There were, however, substantial differences in the rate and direction of change for boys and 
girls who received mental health services (see Figure 1). On the mathematics test, the performance of 
boys increased (from 27% to 39% at or above standard) while the performance of girls decreased 
(from 55% to 43%). The cumulative effect of these changes, however, resulted in similar 10th grade 
performance on the mathematics test for boys and girls who had received services. Performance on the 
English test decreased between 8th and 10th grade for both boys and girls in the treatment group, but 
the decrease for boys (from 48% to 27%) was much greater than the decrease for girls (from 67% to 
56%). There were substantial differences among regions of the state in the amount and direction of 
change in school performance. 

Finally, there were statistically significant differences in the likelihood of criminal justice 
involvement for each of the three groups under examination. During the year after the test, 13% of 
the young people who did not participate were charged with a crime, compared with less than 4% of 
the young people who scored at or above the standard on the test. Boys were more likely than girls to 
be charged with a crime in all three groups, and the difference in rates of criminal justice involvement 
between groups was greater for boys than for girls. 
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Figure 1 

Change in School Performance Between 8^ and 10^ Grade 
for MH Service Recipients by Gender 




Mathematics Skills English Reading 

Discussion 

We believe this pilot study has demonstrated the value of educational test scores for understanding 
and evaluating systems of care for children and adolescents. This rich new source of data can provide 
valid and reliable information on levels of school participation and school performance for recipients 
of mental health and other childrens services programs. This pilot project has highlighted two 
substantive findings that deserve special attention. First, the fact that gender differences in the school 
performance of mental health service recipients are different than those for other students suggests the 
need for careful attention to the impact of gender on the relationship between treatment variables and 
outcomes. 

The use of educational test scores demonstrated here could easily be applied to other service sectors 
and geographical areas. Other service sectors could include child protection, juvenile justice, and 
special education programs. Other geographical areas should include urban areas and regions with 
race/ethnically diverse student populations. Future research would also benefit from consideration of 
the impact of both community characteristics (socio-economic, population density and diversity, etc.) 
and individual student characteristics (both clinical and demographic) on levels of school participation 
and performance for recipients of mental health services and others. 

Finally, future research should consider the impact of treatment modalities and service system 
characteristics on levels of school participation and performance. Differences between emerging 
evidence-based practice and other modalities should be explored. The impact of medication on school 
participation and performance is a particularly important area for investigation. Finally, the impact of 
service system characteristics such as caseload integration and service coordination on school 
participation and performance of service recipients should be the focus of large-scale cross-region 
research. 

Fortunately, the proliferation of electronic databases in conjunction with statistical technologies 
such as Probabilistic Population Estimation provide the opportunity for economical and effective 
research in all of the areas discussed above. This combination of data and methodology provides for 
the exploration of relationships between treatment variables and a variety of important treatment 
outcomes while protecting the confidentiality of medical records and the personal privacy of 
individuals (Pandiani, Banks & Schacht, 1998). This methodology is particularly valuable for 
examining longer-term outcomes and for research that goes beyond the more traditional focus on 
negative outcomes such as hospitalization and incarceration to include positive indicators such as 
employment and participation in post secondary education. 
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